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About  PDT 


•  Research  and  Development  Services 

•  Technologies 

•  Multimodal  Integrated  Bio-Sensing 

•  Biomedical  Signal  Processing  and  Analysis 

•  Custom  Algorithm  Development 

•  Applications 

•  Human  Performance  Optimization 

•  Human-System  Integration 

•  Direct  Neuro-  Control  and  Biofeedback  Systems 


About  this  Contract 


Two  Related  Proposals  &  Projects 

1 .  “Neurosensory  Optimization  of  Information  Transfer 
(NOIT)” 

•  ARO  Proposal  No.  56469-LS 

•  Three-year  basic  research  with  UCLA  team 

2.  “EEG-guided  Input  Lateralization  and  Hemispheric 
Activation  with  Neurofeedback  for  Display  Data 
Control  and  Apprehension.” 

•  ARO  Proposal  No.  59502-LS 

•  One-year  Infrastructure  technology  transfer  to  ARL 


Outline 


•  Problems:  stress,  fatigue,  inattention,  task  complexity 

•  Response:  Neuroergonomic  models  and  control  systems 

-  Create  useful  definitions  of  cognitive  states 

-  Model,  estimate  and  control  cognitive  states 

•  Background  (2003-2009) 

-  Multimodal  sensor-state  models  using  PLS,  KPLS,  KPLS-SVC 

-  Successful  applications:  BCI,  Fatigue,  Engagement,  Overload 

•  New  directions  (2009-201 1 ) 

-  Truly  multidimensional  sensor-process  models 

-  Application  of  Multi-way  methods  (PARAFAC) 

•  Summary 
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Inattentive:  resting,  distracted, 
complacent,  bored 

Engaged:  attending  to  primary 
task  with  minimal  effort  (waiting) 

Attentive:  attention  and  effort 
committed  to  primary  task 

Overioad:  task  demands  exceed 
mental  speed  or  skill  level 

Fatigued:  desire  to  with-draw 
attention  and  effort  from  a  task; 
reduced  executive  control 
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•  the  outermost  layers  of  brain 

•  2-4  mm  thick  (human) 
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other  Elements  of  Sensor-State  Models 


Modality 

Effect  of  Workload 
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Application  1:  Mental  Fatigue 


Black  =  Alert  Red  =  Mentally  Fatigued 


Frequency  (Hz) 


Robust  EEG-Based  Classification  of  Mental  Fatigue 
. 2300  (ba^l)  VS.T906  Mrs  (Day  2) . 
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Application  2:  BCI 
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Application  2:  BCI 
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Control  System  for  Target  Practice 
Trial-by-trial  classification  (left,  right) 

-  250  ms  display  update 
Dual  adaptive  controller  design 

-  Adaptive  PLS  pattern  recognition 

-  Adaptive  gain  control  for  motion 


Application  2:  BCI 


Application  3:  Detection  of  Cognitive  Overload 
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Experiment  for  Overload  Detection 
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Familiar  (bilinear)  Mapping  Algorithms 


Factor  Analysis 


Principal  Component  Analysis  (PCA) 


Multimodal  Mapping 

How  to  generalize  bilinear  models  to  systems  with  more  dimensions? 

1.  Unfolding  a  bilinear  model 

a.  Represent  all  experimental  factors  in  one  dimension 

b.  Observations  (trials)  is  second  dimension 

c.  Contrast  each  dimension  vs.  pairs  of  the  other  two 

2.  Multidimensional  model 

a.  Assume  orthogonal  factors:  PARAFAC 

b.  Allow  interacting  factors:  Tucker  3 

3.  Modeling  approach 

a.  Unsupervised  extraction:  PARAFAC,  CANDECOMP,  Tucker  3 

b.  Supervised  extraction:  N-PLS 
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EEG-based  Estimation  of  Task  Demands 
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Multidimensional  Modeling 

(Tuckers  Model,  unsupervised) 
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•  Xjj^  is  an  element  of  (I  x  m  x  n)  multidimensional  array 

•  F1,  F2,  F3  are  the  number  of  components  extracted  on  the 
1st,  2nd  and  3rd  mode 

•  a,  b,  c  are  elements  of  the  A,  B,  C  loadings  matrices  for  the 
1st,  2nd  and  3rd  mode 

•  g  are  the  elements  of  the  core  matrix  G  which  defines  how 
individual  loading  vectors  in  different  modes  interact 

•  Gijf^  is  an  error  element  (unexplained  variance) 


PARAFAC 

(Parallel  Factor  Analysis,  unsupervised) 


PARAFAC  is  a  special  case  of  the  Tucker  3  model 
where  F1=  F2  =  F3=F  and  G  =  I 
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“Atomic”  EEG  Elements 
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“Molecular”  EEG  Processes 


PARAFAC  Applied  to  U AV  Task 


PARAFAC  EEG  power  model  during  UAV  task  performance  in  one  participant. 
Blue  areas  =  low  demands  (cruise);  red  areas  =  high  demands  (UAV  control). 
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2.  Atom 


1 .  Atom 


Details  of  Atom  1 
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PARAFAC  Applied  to  U AV  Task 


PARAFAC  EEG  coherence  model  during  UAV  task 
performance  in  one  participant 
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Top  10%  of  inter-electrode  coherences 


r  ,,  PARAFAC  Applied  to  Fatigue 

PARAFAC  power  model  during  first  and  last  15  minutes  of  a  3-hour  mental 
arithmetic  task  performance  in  one  participant.  Atom  1  (blue)  did  not  change 
over  time.  Atom  2  (red)  reflected  the  development  of  mental  fatigue. 


Application  Summary 

•  Models  of  engagement,  fatigue  &  BCI 

•  PLS  models  90-100%  accurate 

•  Stable  within  a  day 

•  Stable  from  day  to  day 

*  Detection  of  cognitive  overload 

•  PLS  models 

•  60-90%  accurate 

•  Unstable  over  time 

•  Require  individual  models 

•  PARAFAC  models 

•  Accurate  for  UAV  task,  fatigue 

•  Stable  over  time 

•  Evidence  for  normative  models 


Recommended  Directions 


1 .  Improved  deployable  multimodal  sensors  (EEG, 
fNIR,  EOG,  gaze,  HRV,  EMG,  SCR,  Sp02,  BP,  core  body 
temperature,  gesture,  posture,  facial  expression, ...) 

2.  Multimodal  experimental  designs  and  operational  tests 

3.  Improved  neurocognitive  process  models 

4.  Improved  sensor-process  mapping  algorithms 


Summary 


Successes  and  Failures 

•  Fatigue,  BCI,  engagement:  accurate,  stable 

•  Overload  detection:  variably  accurate,  stable 

New  models  of  state-related  EEG  sources 

•  “Atomic”  EEG  sources 

•  “Molecular”  EEG  systems 

New  multidimensional  models  and  algorithms 

•  Traditional  bilinear  methods  (PCA,  ICA,  PLS,  KPLS) 

•  Truly  multidimensional  methods 

•Correlated  factors  (Tucker  3) 

•Uncorrelated  factors  (PARAFAC,  CANDECOMP,  N-PLS) 
•Supervised  algorithms  (N-PLS) 


